Good music recommenders should not only suggest quality recommendations, but should also allow users to discover new/niche music. User studies capture explicit feedback on recommendation quality and novelty, but can be expensive, and may have difficulty replicating realistic scenarios. Lack of effective offline evaluation methods restricts progress in music recommendation research. The challenge is finding suitable measures to score recommendation quality, and in particular avoiding popularity bias, whereby the quality is not recognised when the track is not well known. This paper presents a low cost method that leverages available social media data and shows it to be effective. Not only is it based on explicit feedback from many users, but it also overcomes the popularity bias that disadvantages new/niche music. Experiments show that its findings are consistent with those from an online study with real users. In comparisons with other offline measures, the social media score is shown to be a more reliable proxy for opinions of real users. Its impact on music recommendation is its ability to recognise recommenders that enable discovery, as well as suggest quality recommendations.

In this article, we'll describe the evolution of our recommendation engine, from the initial minimum viable product (MVP) built with Apache Mahout, to a hybrid offline online pipeline. We'll explore the impact these changes have had on product metrics and how we've addressed challenges by using incremental modifications to algorithms, system architecture, and model format. To close, we'll review some related lessons in system design that apply to any high-traffic machine learning application. Indeed's production applications run in many data centers around the world. Clickstream data, and other application events from every data center, are replicated into a central HDFS repository, based in our Austin data center.